> 



o 



The hidden subgroup problem and permutation group theory 

Julia Kempe Aner Shalev 

UC Berkeley, Berkeley, CA94720 and Institute of Mathematics 

CNRS & LRI, Universite Paris-Sud The Hebrew University 

91405 Orsay Cedex, France Jerusalem 91904, Israel 

;_i. ', kempe@lri.fr shalev@math.huji.ac.il 

O 

O ; February 1, 2008 

(N 

^ . 

Abstract 

OO . We employ concepts and tools from the theory of finite permutation groups in order 

to analyse the Hidden Subgroup Problem via Quantum Fourier Sampling (QFS) for 
the symmetric group. We show that under very general conditions both the weak 
\^ I and the random-strong form (strong form with random choices of basis) of QFS fail to 

^^ ■ provide any advantage over classical exhaustive search. In particular we give a complete 

^D I characterisation of polynomial size subgroups, and of primitive subgroups, that can 

be distinguished from the identity subgroup with the above methods. Furthermore, 
assuming a plausible group theoretic conjecture for which we give supporting evidence, 
f~^ ■ we show that weak and random-strong QFS for the symmetric group have no advantage 

" ^ ' whatsoever over classical search. 

^ ! 1 Introduction 

^ . 

^ , . In the last decade quantum computation has provided us with powerful tools to solve prob- 

lems not known to be classically efhciently solvable, like factoring |Sho94j and discrete log 
|Kit95j . Nearly all the problems in which a quantum computer excels more than quadrati- 
S^ ' cally with respect to its classical counterpart can be cast into the framework of the Hidden 

Subgroup Problem (HSP). Let G be a finite group and H < G a subgroup. Given a function 
/ : G — > 5 that is constant on (left)-cosets gH of H and takes different values for different 
cosets, determine a set of generators for H. The decision version of this problem is to 
determine whether there is a non-identity hidden subgroup or not. 

The reason that quantum computers seem to provide a speed-up for this type of problem 
is that it is possible to implement the Fourier transform over certain groups efficiently on 
a quantum computer. This in turn allows to sample the Fourier components efficiently 
(this technique is referred to as the "standard method"). In the case of Abelian groups G 
(appearing in factoring and discrete log) the hidden subgroup can be reconstructed with 
only a polynomial (in log \G\) number of queries to the function and a polynomial number 
of measurements (samplings in the Fourier basis) and postprocessing steps. 

Addressing the HSP in the non- Abelian case is considered to be one of the most im- 
portant challenges at present in quantum computing. A positive answer to the question 
whether quantum computers can efficiently solve the Hidden Subgroup Problem over non- 
Abelian groups would have several important implications for the solution of problems in 
NP, which are neither known to be NP-complete nor in P; and which are good candidates 
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for a quantum speed-up. Among the most prominent such problems is Graph Isomorphism, 
where the group in question is the symmetric group. 

For many non-Abehan groups it is possible to implement the Fourier transform on a 
quantum computer efficiently EH99i ,RB98, PRB99. GSVVnTl IMHl^Mj . and in particu- 
lar explicit constructions exist for the symmetric group Sn |Bea97j . This fact and the 
prominence of the problems involved make it very desirable to get a handle on the power 
of Quantum Fourier Sampling (QFS) to solve the HSP or its decision version for general 
groups. 

In this paper we focus on the question which hidden subgroups can be distinguished 
from the identity via QFS with special attention to the symmetric group. Several partial 
positive results have been obtained previously for groups that are in some ways "close" to 
Abelian, like some semidirect products of Abelian groups |EH99[ IRB981 |Kup031 IMRRS04J . 
in particular the Dihedral group; Hamiltonian groups |HRT00j . groups with small commu- 
tator groups |IMS01j and solvable groups of constant exponent and constant length derived 
series |FIM"'"03| . Often in these cases the irreducible representations are known and can 
be analysed in a relatively straightforward way. For instance the Dihedral group Dn, the 
first non- Abelian group to be analysed in this context by Ettinger and Hoyer |EH99j . is 
"nearly" Abelian in the sense that all of its irreducible representations (irreps) have degree 
at most two. Indeed hidden reflections of Dn can be distinguished from identity with only 
polynomial Quantum Fourier Samplings, similar to the Abelian case (where all irreducible 
representations are one-dimensional).^ 

The holy grail of the field is the symmetric group Sn, which seems much harder to 
analyse, partly because to this day there is still only partial explicit knowledge about its 
irreducible representations and character values |Sag01| , because most of its subgroups 
are far from normal (have many conjugate subgroups), because most of its irreducible 
representations have very large dimension (2®("i°g"-)) and the number of different irreducible 
representations is an exponentially small fraction of the size of the group, to name just some 
of the difficulties. The structure of distinguishable versus indistinguishable subgroups of Sn 
has remained highly elusive. 

In this work we provide a substantial step towards a complete classification of subgroups 
of the symmetric group for which the decision version of the HSP can be solved efficiently 
via Quantum Fourier Sampling. We bring into play classical notions and results from the 
theory of finite permutation groups, which have not been employed before in quantum 
computing, and seem to be relevant in these investigations. These include notions such as 
minimal degree, rank, subdegrees, and primitivity, which played a key role in permutation 
group theory since the days of Jordan in the 19th century. Moreover, recent advances in 
finite permutation groups, due to sophisticated work by Babai ^Bab8H IBab82j and others 
on the one hand, and due to the Classification of Finite Simple Groups (CFSG) on the 
other hand (see Cameron |(]am81j ). provide us with very powerful machinery. 

Using these notions and machinery we present several new results, which incorporate 
existing results in the area, and we provide a toolbox for further investigations. We are able 
to give both upper bounds and, for the first time in this context, lower bounds on the total 
variation distance of the relevant distributions, and to derive many applications. 



^Note, however, that the computational version of HSP seems much harder: even though a polynomial 
number of samples suffice to distinguish hidden reflections information theoretically, no efficient reconstruc- 
tion procedure beyond sophisticated exhaustive search is known. 



Outline of results: In a nutshell, our main results show that under various conditions 
on the hidden subgroup H < Sn, the following statement is true for the hidden subgroup 

H < Sn'- 

^ H can be distinguished from the identity subgroup with either the weak standard method 
or the strong standard method with random basis only if it contains an element of constant 
support (i.e. a permutation in which all but a constant number of points are fixed). 

Since there is only a polynomial number of such elements in Sn, the statement implies 
that the standard method both in its weak and strong form with random basis provides no 
advantage over classical exhaustive search. 

• Statement ^ is true if H is of polynomial size. 

• It is also true for an important class of subgroups known as primitive groups. These 
subgroups, which can be superpolynomial in size, are considered the building blocks 
of permutation groups. 

• We exhibit a family of even larger subgroups, of exponential order, for which State- 
ment ^ is true. 

The cases we study seem to suggest that Statement ^ might hold for all subgroups H. 
Surprisingly we show that this is indeed true under a plausible group theoretic conjecture, 
for which we provide further evidence. Assuming the conjecture, it follows that Quantum 
Fourier Sampling (with random basis) provides no advantage whatsoever over classical 
exhausitve search. 

Main results: We focus on the weak form of the standard method (see Section [21), since 
the strong form with random choices of basis does not provide any non-negligible additional 
information for the symmetric group and the subgroups we consider |GSVVOT| ^. In partic- 
ular we are not aware of any cases where there is a good basis for the strong method, but 
a random choice of basis does not also solve the HSP. 

To state our main results, let G be a finite group, H < G a subgroup, and let Dh denote 
the total variation distance between the distributions on the irreducible representations of 
G induced by H and by the identity respectively, when sampled with the weak standard 
method. We say that H is distinguishable (using the weak standard method) if Dh > 
(log IGj)^*^ for some constant c, and is indistinguishable otherwise. 

Our starting point is a general result providing both upper and lower bounds on the 
total variation distance Dh in terms of the same group theoretic data. 

Theorem 1.1 Let Gi, . . . ,Gk denote the non-identity conjugacy classes of G. Then 

k k 

Y, \a n H\^\H\-'\Gi\-' <DH<Y,\C^rl H\\C,\-'2. (1) 

1=1 i=l 



^The strong standard method sometimes provides substantially more information than its weak coun- 
terpart, and is indeed necessary to efficiently solve HSP in the case of groups like the Dihedral group 
|EH99I [RupOSI and other semidirect product groups IMRRS041 . An irrep, and hence the non-Abelian QFT, 
is given only up to the choice of basis. Grigni et al. show that for a random basis the additional information 
provided by the strong method is exponentially small, provided the group is sufficiently non-Abelian and 
the hidden subgroup sufficiently small, as is the case for all groups we analyse here. It remains to be seen 
whether judicious choices of basis for each irrep can give more information in the case where random choices 
don't help; but this is believed to be unlikely. 



Applying the upper bound with \H\ = 2 gives the result obtained previously by Hallgren et 
al. and Grigni et al. |HRT001 IGSVVOT] . No lower bounds seem to exist in the literature. 
This theorem has a wide range of applications. For example, it enables us to characterise 
distinguishable subgroups H < G of polylogarithmic order (see Theorem 13.21 below) . 

Specialising to G = Sn we show that the minimal degree of i? is a crucial notion in the 
study of the distinguishability of H. The minimal degree m{H) of a permutation group H is 
defined to be the minimal number of points moved by a non-identity element of H. In other 
words, for g ^ Sn let fix{g) be the number of fixed points of g, and let supp{g) = n — fix{g) 
be the support of g. Then 

m{H) = Ta.m.{supp{h) : 1 7^ /i € //}. 

This notion goes back to the 19th century, and plays an important role in the theory 
of finite permutation groups since the days of Jordan |Jor73L Por75j . It is intriguing that 
it plays some role in the HSP as well, giving a complete characterisation of distinguishable 
polynomial size subgroups: 

Theorem 1.2 Let H < Sn with \H\ < rf for some constant c. Then H is distinguishable 
if and only if its minimal degree m[H) is constant. 

For instance we cannot distinguish a group generated by a cycle of non-constant length 
or an involution with non-constant number of transpositions (implying the result in jHR TflCH 
KtSVVDT] ) . Note that the strength of this theorem comes from the "if-and-only-if " : The 
distinguishable subgroups must contain an element of constant support. Since there are 
only polynomially many such elements in Sn we can just exhaustively query them. 

This also has implications for the Graph Isomorphism (GI) problem. Recall that to solve 
GI for two graphs Gi , G2 , it suffices to distinguish a hidden subgroup of the automorphism 
group Aut{G\ U G2) of the form Hi x H2 (not Gi ~ G2), where Hi = Aut{Gi), from a 
subgroup of the form H U aH {Gi ~ G2), where H = Hi x H2 and a maps Gi to G2 (see 
e.g. I.TozOOi ) . If the automorphism group of each graph is of polynomial size our results 
imply that we cannot distinguish each of the two possible cases from identity, and hence 
(using the triangle inequality) we cannot distinguish them from each other unless Aut(Gi) 
contains an element of constant support. Thus QFS provides no advantage here. 

Our next result concerns primitive subgroups. A permutation group is called primitive 
if it is transitive (has only one orbit) and does not preserve a non-trivial partition (block 
system) of the permutation domain. Primitive permutation groups are considered the build- 
ing blocks of finite permutation groups in general, and were extensively studied over the 
past 130 years. We note that ii H < Sn is primitive and H 7^ An,Sn then Babai showed 
that m{H) > {^Jn - l)/2 and \H\ < n''v^'°g". Using the Classification of Finite Simple 
Groups the latter bound can be somewhat improved to |i?| < 2n^, which is essentially 
best possible |Gam81j : in particular the order of H can be much more than polynomial, 
and so Theorem 11.21 above does not apply. 

However, we obtain the following somewhat surprising general result: 

Theorem 1.3 Let H 7^ An,Sn be a primitive subgroup. Then H is indistinguishable. 

As the hidden subgroups get large we would suppose that it becomes easier to distinguish 
them from the identity. However, we show below that H can get extremely large and yet 
cannot be distinguished with the weak standard method: 



Theorem 1.4 Let e{n) be a sequence of real numbers which tend to zero as n ^ cxd. Then 
for all sufficiently large n there is an indistinguishable subgroup H < Sn of size \H\ > 

I Q \e{n) 

In particular, there are indistinguishable subgroups H of exponential order. 

To prove Theorems 11.31 11.41 we give a somewhat technical group theoretic criterion for 
indistinguishibility of subgroups of non-constant minimal degree (Proposition I4.1() . We 
conjecture that this criterion applies universally. This implies that every distinguishable 
subgroup has a non-identity element of constant support. 

It is interesting that permutation-theoretic data is relevant to the distinguishability 
problem even when the group G in question is not Sn, but an arbitrary finite group. Indeed, 
given H < G there is a standard way to view G (or G/N where A^ is the normal core of 
H) as a transitive permutation group on the set X = G/H of (right) cosets of -ff in G 
(where g G G acts by right multiplication). Recall that a suborbit of G in this action is 
an orbit of H on X, and the rank rx{G) of G is defined to be the number of suborbits of 
G. The subdegrees of G are the sizes of its suborbits. Thus the average subdegree of G is 
\G : H\/rx{G). Setting H^ = gHg~^ it is easy to see that the subdegrees of G have the 
form \H : H n H^\ for g £ G. Using this data, Theorem II. H and classical permutation 
theoretic tools, we obtain the following positive result. 

Theorem 1.5 Suppose \H\ is not polylogarithmic, but the average subdegree of G on G/H 
is polylogarithimic. Then H is distinguishable. In particular this holds when \H : HCiH^l < 
{log\G\rforallgeG. 

This theorem extends the result of IHRTOOl showing that if H is normal in G then H 
is distinguishable (since in this case \H : H f] H^\ = 1 for all g). It also implies the easy 
observation that subgroups of size at least |G|/(log jGI)'^ are always distinguishable. 

Our methods also allow us to examine the more general case of distinguishing between 
two subgroups H and K of G, see Section 3 for some details. 

Further Related Work: The HSP plays a central role in most known quantum algo- 
rithms and the efficient algorithm for the Abelian case using Fourier Sampling is folklore. 
The non- Abelian HSP has received a lot of attention in recent years, due to its connection 
to several candidate problems in NP like Graph Isomorphism (for the symmetric group) 
and lattice problems |Reg02| (for the Dihedral group); we mention only the work relevant 
to ours. 

Desp ite a lot of progress for various non- Abelian groups |EH991 lRB98l IMRRi^Oil IIMSOII 
|FIM"'"03J the results on the symmetric group are very sparse. Grigni et al. |GSVVOT] 
show that sampling the row index in the strong standard method provides no additional 
information. They also show that the additional information provided by the strong method 
in a random basis scales with y^\H\'^k{G)/\G\ where k(G) is the number of conjugacy classes 
of the group G and \H\ the size of the hidden subgroup. Both Hallgren et al. and Grigni 
et al. |HRT00l iGSVVOl] show that hidden subgroups of Sn of size \H\ =2, generated by 
involutions with large support, cannot be distinguished from identity; exactly the task that 
needs to be solved for Graph Automorphism. 

Hallgren et al. |HRT00j also point out that the weak standard method cannot distinguish 
between conjugate subgroups. In ERTOOl IGSVVOi] it is shown that the weak standard 
method allows us to efficiently determine the normal core of a hidden subgroup H and hence 
in particular normal subgroups. 



2 Preliminaries and notation 

Fix a finite group G and a subgroup H < G. We denote states of tlie vector space C[G], 
spanned by the group elements, witli a |-), as is standard in quantum computation^. 

The Quantum Fourier Transform (QFT) over a group G is the following unitary trans- 
formation on C[G]: 



\9) -^ -^f^2Z^pP^3)ij\p,hJ) 



1 v^ 

where p labels an irreducible representation of G, dp is its dimension and 1 <i,i < dp. The 
\p,i,j) span another basis of C[G], the so called Fourier basis. 

The standard method of Quantum Fourier Sampling is the following: The state is ini- 
tialised in a uniform superposition over all group elements; a second register is initialised to 
|0). Then the function / is applied reversibly over both registers (i.e. / : \g)\0) — > \g)\fig)))- 
Finally the second register is measured, which puts the first register into the superposition 
of a (left)-coset of H, i.e. in the state \gH) := .| X^heH 1^^) ^°^ some random g ^ G. 



Finally the QFT over G is performed, yielding the state 

/r^Wm Y^y^pYl Pij(9h)\p, i,j). 

A basis measurement now gives {p,i,j) with probability PgH(p,z,j) = i ^if^i | J2heH Piji9^)\'^ ■ 
Since we do not know g and g is distributed uniformly, we sample {p, i,j) with probability 
^^ ~ IgT So ^gH- The strong standard method samples both p and its entries i,j. In the 
weak standard method only the character Xp is measured (but not the entries i,j, which 
are averaged over)"^. The probability to measure p in the weak case is 

PH{p) = ^^^Xp{h). 
' ' h&H 

Let Irr{G) be the set of irreducible characters of G. Then Ph is a distribution on Irr(G). 
To solve HSP we need to infer H from the resulting distribution. Distinguishing the 
trivial subgroup {e} from a larger subgroup H efficiently using the standard method is 
possible if and only if the Li distance Dh between Prgi and Ph is larger than some inverse 
polynomial in log |G|. The Li distance (also known as the total variation distance) is given 
as 

We say that H is distinguishable (using the weak standard method) if Dh > (log|G|)~'^ 
for some constant c, and indistinguishable otherwise. If K < G is another subgroup we let 
D{H,K) = \Ph — Pk\i be the Li distance between the distributions Ph and Pk- 

We also need some group theoretic notation. For x G G we let x*^ denote the conjugacy 
class of X in G. Let Ci,...,Cfc denote the non-identity conjugacy classes of G. For an 
irreducible character Xp ^ Irr{G) we let xp{Ci) denote the common value of Xp{x) for 
elements x ^ Ci. 



^For the necessary background in quantum computation see e.g. |NCOO| . 

*It is easy to sec HRTOO GSVVOl^ that for the weak standard method the probabihty to sample p is 



independent of the cosct of H wc happen to land in. 



3 Arbitrary groups 

In this section we discuss results for arbitrary groups G, providing some proofs when space 
aUows. 

Proof of Theorem \1.1\ For each irreducible representation p of G we have 
I Yl ^p(^)l^ E \Xpih)\< E d,<\H\dp. 

h(^H,hy^e h&H,h^e h&H,h^e 

Hence dp > \H\^^\ J2heH h^^e^pWl- Substituting this in ((2j we obtain 

' " ' p heH,h=/=e 

Note that xp{h) = XpiQ) if h e HnQ. This yields EheH,h^eXp{h) = Eti \HnGi\xp{Gi), 
and so 

k 
' " ' p i=l 

Now, 

k k 

j=l i=l ij^j 

Using the generalised orthogonality relations we observe that 

k k 

p i=l 1=1 

and 

^5^|i?nQ||i/nc,|xp(Q)xp(C,) = o. 
p i^i 

It follows that 

This completes the proof of the lower bound. To prove the upper bound, write 

DH\G\=Y,d,\ Y. Xp(MI<E^P E \Xp{h)\= E Y.dp\Xp{h)\. (3) 

P heH,h^e p heH,h^e heH,h^e p 

Fix h £ H and choose z such that h £ Ci. Using the Cauchy-Schwarz inequality we 
obtain 

Y.dp\Xpih)\<{j:dy"(Y.\Xpih)n^^ 
p p p 

giving (using the orthogonality relations) Epdp\Xp{h)\ < |G|V2(|G|/|Ci|)V2 = |G||Q|-i/2. 
Summing over non-identity elements h £ H, and observing that the upper bound above 
occurs \Hr\Ci\ times, we obtain Ylh€H,hfte^pdp\Xp{h)\ < J2i^i \Hr\Gi\\G\\Ci\~'^/'^. Com- 
bining this with (jSJ we obtain Dh < ^j^j^ \H n Cj||Cj|~^'^, as required. D 
The following is an immediate consequence of Theorem 11.11 



Corollary 3.1 Let Cmin denote a non-identity conjugacy class of minimal size intersecting 
H non-trivially . Then we have 



\H\ \Cmin\ < Dh < {\H\ - l)\Cmin\ 



"1/2 



We can now characterise distinguishable subgroups of polylogarithmic order in an arbi- 
trary group G. Assuming \H\ is polylogarithmic Corollary 13. II shows that Djj is polyloga- 
rithmic if and only if \Cmin\ is- In other words we have proved the following. 

Theorem 3.2 Suppose \H\ < (log|G|)'^ for some constant c. Then H is distinguishable if 
and only if H has a non-identity element h such that \h'^\ < (loglGI)"^ for some constant 
d. 

There is an interesting reformulation of the lower bound in Theorem 11.11 in terms of 
fixed points. Regard G as a permutation group on X = G/H. Denote by fixx{g) the 
number of fixed points of 5 G G in this action. Let r = rx{G) denote the rank of G in this 
action, namely, the number of orbits of the point stabilizer H on X. 

Corollary 3.3 With the above notation we have 
(i)DH > ^T.h&H,h^ef^^x{h); 
(ii)DH>rx{G)/\X\-l/\H\. 

Proof: It is weU known that, if g G C,, then fixx{g)/\X\ = \H r\Ci\/\Ci\. 
Therefore 

k 

Y,\Hr\Ci\''\H\~^\C,[~^ = \H\-'^ J2 fixx{h)/\X\ = \G\~' ^ fixx{h). 

i=l h(^H,hj^e h&H,hj^e 

Combining this with Theorem II. II we deduce part (i). 

To prove part (ii) we use the well known lemma of Frobenius that the number of orbits 
of iJ on X equals the average number of fixed points ol h € H on X (see for instance 
Theorem 1.7A of _DM96j). This shows that 

\H\-' Y, fixx{h)=rx{G)-\X\/\H\. 

h(^H,hj^e 

Dividing both sides by \X\ we deduce part (ii) from part (i). D 

Proof of Theorem \1.5\ This follows easily from part (ii) of the above Corollary. D 

We close this section by considering the more general problem of distinguishing between 
two arbitrary subgroups H and K of G. Obviously, if the total variation distance D{H, K) 
between the respective distributions is zero then the weak standard method cannot distin- 
guish between H and K, even if super polynomial complexity is allowed. This gives rise to 
the fundamental problem of characterising subgroups H, K of distance zero. It has already 
been observed that conjugate subgroups have distance zero ^HRTOOj . does the converse 
hold? 

To solve these problems, recall that the permutation representation of G on G/H gives 
rise to a linear representation of G in dimension G/H (which can be realised using the 
corresponding permutation matrices). We can now state 



Theorem 3.4 The following are equivalent for subgroups H,K < G. 

(i) D{H,K) = 0. 

(a) For each conjugacy class C of G we have \H riG\ = \K riG\. 

(Hi) The permutation representations of G on G/H and G/K give rise to equivalent 
linear representations. 

Moreover, there exist finite groups G and non-conjugate subgroups H,K < G such that 
D{H,K) =0. 

Our proof of the equivalence of (i)-(iii) is elementary, based on the fact that the char- 
acters in Irr(G) form a base for the class functions on G. The proof of the last assertion is 
deeper and will be omitted in this version. 

4 Symmetric groups 

Let us now focus on the case G = Sn- 

Proof of Theorem MJA. Let g £ Sn with supp{g) = k. Then it is straightforward to verify that 
(^) < Id \ <n- As a consequence we see that a conjugacy class G in Sn has polynomial 
order if and only if it consists of elements of constant support. This observation, when 
combined with Theorem \\\.'A completes the proof of Theorem 11.21 D 

The proofs of Theorems 11.(^1 and 11.41 are longer and less elementary, and so we will only 
sketch them here. In the heart of both proofs lies the following somewhat technical result. 

Proposition 4.1 Let H < Sn be a subgroup with non-constant minimal degree. Suppose 
that, for each k < n, H has at most n^'"^ elements of support k. Then H is indistinguishable. 

Proof: Apply the upper bound of Theorem ll.il written in the form 

To evaluate this sum we use a result from |LSnij . showing that, for G = Sn and for h £ G 
of support k we have |/i | > n for any real number a < 1/3 and n large enough (given a). 
Setting 

Hk = {h £ H : supp{h) = k}, 

we obtain 

Dh< Y. \Hk\n-'\ 

k>m{H) 

for any real number b < 1/6 and sufficiently large n. Fix b with 1/7 < b < 1/6, and set 
c = b — 1/7, m = m{H). Then 



Dh<Y n^'^n-^^ = Yl ^^^'^ - ^^" 



Since m = m{H) is non-constant, we see that Dh is smaller than than any fixed negative 
power of n, and so H is indistinguishable. D 



Now, for Theorem 11.31 we use Babai's lower bound on the minimal degree of primitive 
subgroups H ^ An, Sn |Bab81j . showing that 

m(iJ) > (^-l)/2. (4) 

Furthermore, we apply a theorem of Cameron |Cam81j (which in turns relies on the Clas- 
sification of Finite Simple Groups) describing all primitive groups of 'large' order. In par- 
ticular it follows from that description that, for all large n, and for a primitive subgroup 
H / An,Sn, either 

(i) \H\ <n'='^'^', or 

(ii) n = (2) for some /, and H < Si acting on 2-subsets of {1, . . . , Z}, or 

(iii) n = l^ for some /, and H < Sil S2 acting on {1, . . . ,/}^ in the so called product 
action. 

We claim that for all large n and for all k we have \Hj^\ < n^'"^ . 

To show this it suffices to consider k > {^/n — l)/2, otherwise \Hk\ = by (HJ. Now, 
if H satisfies condition (i) above then the claim follows trivially using \Hk\ < \H\. So it 
remains to consider groups H in cases (ii) and (iii). Here a detailed computation based on 
the known actions of H, which we omit from this version, completes the proof of the claim. 

At this point Proposition l4.1l can be applied, and we conclude that H is indistinguishable. 
In fact our argument shows that, for some e > 0, all primitive subgroups H 7^ An, Sn satisfy 
Dh < n-'^. 

Finally, to prove Theorem 11.41 we construct H as the full symmetric group on [n/rj 
blocks of size r, where r = r(n) tends very slowly to infinity. Then m{H) = 2r, which is 
non-constant. A detailed computation shows that the second assumption of Proposition 14. II 
also holds, which yields the desired conclusion. The details are left to the reader. 

We end this paper with 

Conjecture 4: Suppose H < Sn is distinguishable. Then its minimal degree m{H) is 
constant. 

Conjecture Jft: Every subgroup H < Sn with non-constant minimal degree has at most 
n '^ elements of support k. 

Proposition 14. II shows that Conjecture il|k implies Conjecture 4|k. We regard Conjecture 
X as a plausible group theoretic conjecture, for which we have mounting evidence. 

First note that by the above discussion Conjecture il|k holds for primitive groups. Sec- 
ondly we can show that the set of subgroups satisfying the conjecture is closed under direct 
products. Thirdly we can prove the conjecture for wreath products KlL, \i K satisfies the 
conjecture. Recall that all transitive imprimitive groups are subgroups of wreath products 
W = Ski Si and the maximal ones are the full wreath product W . 

Hopefully the methods introduced in this paper and the group theoretic reductions will 
lead to a full classification of distinguishable subgroups of Sn and of other groups. 
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